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Abstract. We show that the membership problem in a finitely generated sub- 
monoid of a graph group (also called a right-angled Artin group or a free partially 
commutative group) is decidable if and only if the independence graph (commu- 
tation graph) is a transitive forest. As a consequence we obtain the first example 
of a finitely presented group with a decidable generalized word problem that does 
not have a decidable membership problem for finitely generated submonoids. We 
also show that the rational subset membership problem is decidable for a graph 
group if and only if the independence graph is a transitive forest, answering a 
question of Kambites, Silva, and the second author [24]. Finally we prove that 
for certain amalgamated free products and HNN-extensions the rational subset 
and submonoid membership problems are recursively equivalent. In particular, 
this applies to finitely generated groups with two or more ends that are either 
torsion-free or residually finite. 



1 Introduction 

Algorithmic problems concerning groups are a classical topic in algebra and theoretical 
computer science. Since the pioneering work of Dehn from 1910 [8], decision problems 
like the word problem or the generalized word problem (which is also known as the 
subgroup membership problem since it asks whether one can decide if a given group 
element belongs to a given finitely generated subgroup) have been intensively studied 
for various classes of groups. A first natural generalization of these classical decision 
problems is the submonoid membership problem: given a finite set S of elements of G 
and an element g ^ G, does g belong to the submonoid generated by SI Notice that g 
has finite order if and only if g^^ is in the submonoid generated by g and so decidability 
of the submonoid membership problem lets one determine algorithmic ally the order of 
an element of the group G. A recent paper on the submonoid membership problem is 
Margolis, Meakin, and Sunik [27]. 

A further generalization is the rational subset membership problem: for a given 
rational subset L of a group G and an element g e G it is asked whether g e L. 
The class of rational subsets of a group G is the smallest class that contains all finite 
subsets of G, and which is closed under union, product, and the Kleene hull (or Kleene 
star; it associates to a subset L C G the submonoid L* generated by L). Equivalently, 
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it consists of the all subsets of G recognizable by finite automata. Rational subsets in 
arbitrary groups and monoids are an important research topic in language theory, see, 
e.g., [3,24,31]. The rational subset membership problem generahzes the submonoid 
membership problem and the the generaUzed word problem for a group, because every 
finitely generated submonoid (and hence subgroup) of a group is rational. 

It is easy to see that decidability of the rational subset membership problem trans- 
fers to finitely generated subgroups. Grunschlag has shown that the property of having a 
decidable rational subset membership problem is preserved under finite extensions, i.e., 
if G has a decidable rational subset membership problem and G < H, where the index 
of G in ff is finite, then H also has a decidable rational subset membership problem 
[20]. Kambites, Silva, and the second author [24] proved that the fundamental group 
of a finite graph of groups [36] with finite edge groups has a decidable rational sub- 
set membership problem provided all vertex groups have a decidable rational subset 
membership problem. In particular, this implies that decidabiUty of the rational subset 
membership problem is preserved by free products, see also [31]. 

The main result of this paper is to characterize the decidability of the submonoid 
membership problem and the rational subset membership problem for graph groups. In 
particular we provide the first example, as far as we know, of a group with a decidable 
generalized word problem that does not have a decidable submonoid (and hence rational 
subset) membership problem. 

A graph group [13] /) is specified by a finite undirected graph {S, I), which 
is also called an independence alphabet (or commutation graph). The graph group 
I) is formally defined as the quotient group of the free group generated by S 
modulo the set of all relations ab = ba, where (a, b) G I. Graph groups are a group ana- 
logue to trace monoids (free partially commutative monoids), which play a prominent 
role in concurrency theory [12]. Graph groups are also called free partially commuta- 
tive groups [10, 40], right-angled Artin groups [6, 7], and semifree groups [2]. They are 
currently a hot topic of interest in group theory, in particular because of the richness 
of the class of groups embeddable in graph groups. For instance, the Bestvina-Brady 
groups, which were used to distinguish the finiteness properties Tn and FP„ [4] (and 
were also essential for distinguishing the finiteness properties FDT and FHT for string 
rewriting systems [33]), are subgroups of graph groups. Crisp and Wiest show that the 
fundamental group of any orientable surface (and of most non-orientable surfaces) em- 
beds in a graph group [7]. Another class of groups that embed into graph groups are 
fundamental groups of finite state complexes [16]. 

Algorithmic problems concerning graph groups have been intensively studied in 
the past, see, e.g., [10, 11, 15, 24,25,40]. In [10,40] it was shown that the word prob- 
lem for a graph group can be decided in linear time (on a random access machine). 
A recent result of Kapovich, Weidmann, and Myasnikov [25] shows that if {S, I) is a 
chordal graph (i.e., if 1) does not have an induced cycle of length at least 4), then 
the generalized word problem for G(i7, /) is decidable. On the other hand, a classical 
result of Mihailova [30] states that already the generaUzed word problem for the direct 
product of two free groups of rank 2 is undecidable. Note that this group is the graph 
group G(i7, /), where the graph [S, I) is a cycle on 4 nodes (also called C4). In fact, 
Mihailova proves a stronger result: she constructs a fixed subgroup H of G(C4) such 
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that it is undecidable, whether a given element of G(C4) belongs to H. Recently, it was 
shown by Kambites that a graph group G{IJ, I) contains a direct product of two free 
groups of rank 2 if and only if {S, I) contains an induced C4 [23]. This leaves a gap 
between the decidability result of [25] and the undecidability result of Mihailova [30] . 

In [24] it is shown that the rational subset membership problem is decidable for 
a free product of direct products of a free group with a free Abehan group. Such a 
group is a graph group G{S, I), where every coimected component of {S, I) results 
from connecting all nodes of a clique with all nodes from an edge-free graph. On the 
other hand, the only undecidability result for the rational subset membership problem 
for graph groups that was known so far is Mihailova's result for independence alphabets 
containing an induced C4. 

In this paper, we shall characterize those graph groups for which the rational subset 
membership problem is decidable: we prove that these are exactly those graph groups 
I), where {E, I) is a transitive forest (Theorem 2). The graph {S, I) is a transitive 
forest if it is the disjoint union of comparabihty graphs of rooted trees. An alternative 
characterization of transitive forests was presented in [39] : (17, /) is a transitive forest if 
and only if it neither contains an induced C4 nor an induced path on 4 nodes (also called 
P4). Graph groups G{S,T), where {S,I) is a transitive forest, have also appeared in 
[28]: they are exactly those graph groups which are subgroup separable (the case of P4 
appears in [32]). Recall that a group G is called subgroup separable if, for every finitely 
generated subgroup H < G and every g e G\H there exists a normal subgroup N < G 
having finite index such that g ^ NH. Subgroup separabiUty implies decidabihty of the 
generalized word problem. 

One half of Theorem 2 can be easily obtained from a result of Aalbersberg and 
Hoogeboom [1]: The problem of deciding whether the intersection of two rational sub- 
sets of the trace monoid (free partially commutative monoid) M(i7, /) is nonempty is 
decidable if and only if {S, T) is a transitive forest. Now, L n -ftT ^ for two given 
rational subsets L,K C M{E, I) if and only if 1 e LK'^ in the graph group I). 
Hence, if (E, I) is not a transitive forest, then the rational subset membership problem 
for G{E, I) is undecidable. In fact, we construct a fixed rational subset L C G[E, I) 
such that it is undecidable whether 5 G i for a given group element g £ G{E,I). 

The converse direction in Theorem 2 is an immediate corollary of our Theorem 1, 
which is one of the main group theoretic results of this paper. It states that the ra- 
tional subset membership problem is decidable for every group that can be built up 
from the trivial group using the following four operations: (i) taking finitely generated 
subgroups, (ii) finite extensions, (iii) direct products with Z, and (iv) finite graphs of 
groups with finite edge groups. Note that the only operation that is not covered by the 
results cited earlier is the direct product with Z. In fact, it seems to be an open ques- 
tion whether decidability of the rational subset membership problem is preserved under 
direct products with Z. Hence, we have to follow another strategy. We will introduce 
a property of groups that implies the decidability of the rational subset membership 
problem, and which has all the desired closure properties. Our proof of Theorem 1 uses 
mainly techniques from formal language theory (e.g., semilinear sets, Parikh's theorem) 
and is inspired by the methods from [1,5]. 
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It should be noted that due to the above reduction from the intersection problem for 
rational trace languages to the rational subset membership problem for the correspond- 
ing graph group, we also obtain an alternative to the quite difficult proof from [1] for 
the imphcation "(17, /) is a transitive forest intersection problem for rational subsets 
ofM(r, J) is decidable". 

In Section 4 we consider the submonoid membership problem for groups. We prove 
that for an amalgamated free product G *a H such that ^ is a finite proper subgroup 
of G and H and there exist g £ G, h G H with g~^Ag n A = 1 h~'^Ah fl A, the 
rational subset membership problem is recursively equivalent to the submonoid mem- 
bership problem (Theorem 4). An analogous result is proved for certain HNN exten- 
sions with finite associated subgroups. As a consequence we obtain that the rational 
subset membership problem is recursively equivalent to the submonoid membership 
problem for a group with two or more ends that is either torsion-free or residually finite 
(Corollary 2). Using similar techniques, we are also able to prove that the submonoid 
membership problem is undecidable for the graph group 6(17, 1), where {E, I) is P4 
(Theorem 7). The result of [25] shows that this graph group does have a decidable gen- 
eralized word problem, thereby giving our example of a group with a decidable gen- 
erahzed word problem but an undecidable submonoid membership problem. Together 
with Mihailova's undecidability result for C4 and our decidability result for transitive 
forests (Theorem 2) it also follows that the submonoid membership problem for a graph 
group I) is decidable if and only if [S, I) is a transitive forest (Corollary 3). 

Another consequence of our results is that the rational subset membership prob- 
lem for groups is recursively equivalent to the submonoid membership problem if and 
only if a free product of groups with decidable submonoid membership problems has a 
decidable submonoid membership problem. 

2 Preliminaries 

We assume that the reader has some basic knowledge in formal language theory (see, 
e.g., [3, 22]) and group theory (see, e.g., [26, 35]). 

2.1 Formal languages 

Let 17 be a finite alphabet. We use = {a"^ \ a G S} to denote a disjoint 

copy of S. Let = S U IJ~^. Define (a~^)~^ = a; this defines an involution 
~^ : E"^^ —>■ S"^^, which can be extended to the free monoid by setting 

(oi ■ • • an)~^ = cin^ ■ ■ ■ cii^- For a word w £ S* and a £ E we denote by \w\a 
the number of occurrences of a in w. For a subset T C i7, we denote by iTr{w) the 
projection of the word w to the alphabet F, i.e., we erase in w all symbols from S\r. 

Let be the set of all mappings from E to N. By fixing an arbitrary linear order 
on the alphabet E, we may identify a mapping / G N"^ with a tuple from N'^'. For 
a word w e E*, the Parikh image ^{w) is defined as the mapping ^{w) : i7 ^ N 
such that \i'{w)]{a) = \w\a for all a e E. For a language L C E*, the Parikh image 
is If (L) = {^[w) \ w £ L}. For a set i^T C and T C let Tfr{K) = {f\r £ 
I / e K}, where f\r denotes the restriction of / to F. We also need a notation 
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for the composition of erasing letters and taking the Parikh image. So, for L C E* and 
r C E, M^r{L) =Wr{^{L)){= !f'(7rr(-£'))); it may be viewed as a subset of N'-'^I. 
A special case occurs when F = %. Then either >Z'0(L) = (if L = 0) or ^iij{L) is the 
singleton set consisting of the unique mapping from to N. 

A subset C N'^ is said to be linear if there are x, xi, . . . , S such that 
K = {x + \iX\ + • • • + \(.X(, I Ai, . . . , e N}, i.e. -ftT is a translate of a finitely 
generated submonoid of N*^. A semilinear set is a finite union of linear sets. 

Let G — {N, r, S, P) be a context-free grammar, where N is the set of nontermi- 
nals, r is the terminal alphabet, 5 € is the start nonterminal, and P C Nx{NL}r)* 
is the finite set of productions. For u,v G (NuF)* we write u vifv can be derived 
from u by applying a production from P. For A G N, we define L{G, A) = {w G F* \ 
A 4>G w} and L{G) = L(G, S). Parikh's theorem states that the Parikh image of a 
context-free language is semilinear [34]. 

We will allow a more general form of productions in context-free grammars, where 
the right-hand side of a production is a regular language over the alphabet NlJF. Such 
a production A ^ L represents the (possibly infinite) set of productions {A — > s | s e 
L}. Clearly, such an extended context-free grammar can be transformed effectively into 
an equivalent context-free grammar with only finitely many productions. 

Let M be a monoid. The set RAT(Af) of all rational subsets of M is the smallest 
subset of 2^, which contains all finite subsets of M, and which is closed under union, 
product, and Kleene hull (the Kleene hull L* of a subset i C M is the submonoid of 
M generated by L). By Kleene's theorem, a subset i C Z"* is rational if and only if L 
can be recognized by a finite automaton. If M is generated by the finite set S and h : 
17* ^ M is the corresponding canonical monoid homomorphism, then L 6 RAT(M) 
if and only if L — h{K) for some K G RAT(Z'*). In this case, L can be specified by 
a finite automaton over the alphabet S. The rational subsets of the free commutative 
monoid N*' are exactly the semilinear subsets of N*^ [14]. 

2.2 Groups 

Let G be a finitely generated group and let 17 be a finite group generating set for G. 
Hence, 17=*=^ is a finite monoid generating set for G and there exists a canonical monoid 
homomorphism h : (S^^)* G. The language 

WPj:{G) = h-\l) 

is called the word problem of G with respect to S, i.e., WPe{G) consists of all words 
over the alphabet which are equal to 1 in the group G. It is well known and easy 
to see that if F is another finite generating set for G, then WPi;(G) is decidable if and 
only if WPr(G) is decidable. 

The submonoid membership problem for G is the following decision problem: 

INPUT: A finite set of words A C {S^^)* and a word w G (S^^)*. 
QUESTION: h{w) e h{A*)7 

Note that the subset h{A*) C G is the submonoid of G generated by h{A) C G. If 
we replace in the submonoid membership problem the finitely generated submonoid 
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Fig. 1. Case (3) in Lemma 1: we have ffiffjaSjsfefl''" = 1 in either Gi or G2 

h(A*) by the finitely generated subgroup /),((/! U A^^)*), then we obtain the subgroup 
membership problem, which is also known as the generalized word problem for G. This 
term is justified, since the word problem is a particular instance, namely with zi = 0. 
A generalization of the submonoid membership problem for G is the rational subset 

membership problem: 

INPUT: A finite automaton A over the alphabet S"^^ and a word w G (Z"^^)*. 
QUESTION: h{w) e h{L{A))l 

Note that h{w) G h{L{A)) if and only if 1 e h{w~'^L{A)). Since is again a 

rational language, the rational subset membership problem for G is recursively equiva- 
lent to the decision problem of asking whether 1 e h{L{A)) for a given finite automa- 
ton A over the alphabet S"^^. It should be noted that for all the computational problems 
introduced above the decidability is independent of the chosen generating set for G. 

In the rational subset (resp. submonoid) membership problem, the rational subset 
(resp. submonoid) is part of the input. Non-uniform variants of these problems, where 
the rational subset (resp. submonoid) is fixed, have been studied as well. More generally, 
we can define for a subset S C G the membership problem for S within G: 

INPUT: A word w; e 
QUESTION: h{w) e 5? 

The free group F{IJ) generated by S can be defined as the quotient monoid 

F{S) = (^±i)7{aa-i = £ I a G E^^}. 

As usual, the free product of two groups Gi and G2 is denoted by Gi * G2. We will 
always assume that Gi n G2 = 0. An alternating word in Gi * G2 is a sequence 
5152 • • • 5m with TO > 0, (/i G Gi U G2, and G Gi fji+i G G2. Its length is m. 
The alternating word gig2 • • • 5m is irreducible if gi ^ 1 for every 1 < i < m. Every 
element of Gi * G2 can be written uniquely as an alternating irreducible word. We will 
need the following simple fact about free products: 

Lemma 1. Let gig2 ■ ■ ■ gm be an alternating word in Gi * G2. If 9192 ■ ■ ■9m = ^ in 
Gi * G2, then one of the following three cases holds: 

(1) m<l 

(2) there exists 1 < i < m such that 5152 ■ ■ ■ 9i = 9i+i ■ ■ ■ 9m = 1 in Gi * G2 

(3) there exist i G {1, 2}, k > 2, and ^ = j\ < 32 < ■ ■ ■ < jk = fn such that 

.9jn.%2: ■■■^9ok G gj^gj^ ■ ■ ■ 9jk =1 in Gi, and 9j^+i9j^+2 ■ • -ffj^+i-i = 1 in 
Gi*G2foralll<£ <k. 
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Proof. Case (3) from the Lemma is visualized in Figure 1 for = 5. Shaded areas 
represent alternating sequences, which are equal to 1 in Gi * G2 . The non-shaded blocks 
are either aU from Gi or from G2, and their product equals 1 in Gi or G2, respectively. 

We prove the lemma by induction on m, the case m < 1 being trivial. So assume 
that m > 2. Since gig2 ■ • ■ Qm = 1 in Gi * G2, there must exist 1 < j < m with gj = 1. 
If j = 1 or j = m, then we are in case (2) from the lemma. Hence, we may assume that 
m > 3 and that 2 < j < m - 1. It follows 

91- ■■ 9j-2{9j^i9j+i)9]+2 • • ■ 5m = 1 

in Gi * G2. Since the alternating word gi • • ■ gj-2{gj-igj+i)gj+2 ■ ■ ■ gm has length 
rn — 2, we can apply the induction hypothesis to it. If m — 2 = 1, i.e., m = 3, then 
we obtain case (3) from the lemma (with k = 2, ji = 1, and j2 = 3). If a non-empty 
and proper prefix of • • • gj-2i9j-igj+i)gj+2 ■ ■ ■ gm equals 1 in the group Gi * G2, 
then the same is true for 9192 - ■■ gm- Finally, if case (3) from the lemma applies to 
the alternating word 51 • • • gj-2{9j-i9j+i)gj+2 ■ ■ ■ 9m> then again the same is true for 
9192- ■■ 9m- □ 

Notice that (3) in Lemma 1 can only occur when m is odd. 

Assume that A < G and B < H arc groups and ip : A ^ B ism isomorphism. 
The amalgamated free product G*^ His the quotient 

{G*H)/{a = ip{a) \ a&A}. 

Without loss of generaUty we may assume that A = G f\ H and that is the identity 
map on A; in this situation we briefly write G *a H for G *,p H. Every element of 
G*aH can be written as a word ci • • • c„, where n > 0, ci , . . . , c„ e G U if, if n > 1 
then ci, . . . , c„ e (G U iJ) \ A, if n = 1 then ci ^ 1, and Cj e G \ ^ Ci+i &H\A 
for all 1 < i < n. Such a word is called a reduced sequence. The normal form theorem 
for amalgamated free products states that every nonempty reduced sequence represents 
a nontrivial element of G *aH [26, Chapter IV, Theorem 2.6]. 

If G is a group and ip : A B is w isomorphism between subgroups A, B of G, 
then the HNN extension *y G, with base G, stable letter t, and associated subgroups 
A, B is the quotient group 

G * {t)/{t~^at = (p{a) \ ae A} 

where t is the generator of an infinite cycUc group. Every element of *^ G can be 
written as a word got^^gi ■ ■ ■ t^'^gn, where n > 0, go, ■ ■ ■ ,gn € G, and ei, . . . , e„ G 
{1,-1}. Such a word is referred to as a reduced sequence if it contains no factor of the 
form t~^at or tbt~^ with a & A, respectively b € B. Britton's Lemma [26, Chapter IV] 
says that ifw = got^^gi ■ ■ ■ t^'^gn is a reduced sequence with n > 1, then w represents 
a nontrivial element of *y G. 

We will also consider fundamental groups of finite graphs of groups, which is a 
group theoretic construction generaUzing free products, free products with amalgama- 
tion, and HNN-extensions, see e.g. [36]. We omit the quite technical definition. In order 
to deal with the rational subset membership problem for graph groups, free products 
suffice. 
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2.3 Trace monoids and graph groups 

In the following we introduce some notions from trace theory, see [9, 12] for more 
details. An independence alphabet is just a finite undirected graph {E, I) without loops. 
Hence, / C 17 x 17 is an irreflexive and symmetric relation. The trace monoid M(i7, 1) 
is defined as the quotient 

M{E, I) = E*/{ab = ha \ (a, b) e /}. 

Elements of M{S, I) are called traces. Note that M(r, J) is a submonoid of M(S, I) 
in case {F, J) is an induced subgraph of {S, I). The latter means that F C S and 
J = lr){r X F). 

Traces can be represented conveniently by dependence graphs, which are node- 
labelled directed acyclic graphs. Let m = ai • • • a„ be a word, where € S. The vertex 
set of the dependence graph of « is {1, . . . , n} and vertex i is labelled with e S. 
There is an edge from vertex i to j if and only if i < j and (a^, aj) ^ /. Then, two words 
define the same trace in M(Z', /) if and only if their dependence graphs are isomorphic. 
The set of minimal (resp. maximal) elements of a trace t € M.{E, I) is min(<;) = {a € 
E \3ue M(r, /) : t = au} (resp. max(t) = {a e E \ 3u e M{E, I) : t = ua}). A 
trace rewriting system R over M.{E,F) is just a finite subset of M{E, I)y.M.{E,F) [9] . 
We can define the one-step rewrite relation -^r C M.{E, I) x M.{E, I) by: x — >ij y if 
and only if there are u, w S M(I7, /) and {f.,r) (£ R such that x = iilv and y = urv in 
1^1(^7, /). A trace t is irreducible with respect to R if there does not exist a trace u with 
t -^R u. 

The graph group 'G{E, I) is defined as the quotient 

I) = F{E) /{ah = ha \ (a, 6) e /}. 

If {E, I) is the empty graph, i.e., T = 0, then we set M{E, I) = G,{E, I) = 1 (the 
trivial group). Note that (a, b) £ I implies a~^b = ha~^ in G{E, I). Thus, the graph 
group G{E, I) can be also defined as the quotient 

G{E,I) =M{E'^^,I)/{aa-^ =e\ a € r^^}. 

Here, we implicitly extend ICExEtoIC E^'^ x E^'^ by setting (a", b'') € / if 
and only if (a, b) e I for a,b e E and a, /3 £ {1, -1}. Note that M{E, I) is a rational 
subset of G(i:,/). 

Define a trace rewriting system R over M{E^^,I) as follows: 

R={{aa-^,s)\aeE^^}. (1) 

One can show that for every trace t e M.{E^^,I), there exists a unique normal form 
NFn^t) such that t NFR{t) and NF7^(t) is irreducible with respect to R. More- 
over, for allu,v€ M{E^^,I), u = vm G{E, I) if and only if NFfl(M) = NFr{v) (in 
M(i:='=\ /)) [10]. This leads to a linear time solution for the word problem of G{E, I) 
[10,40]. 

If the graph {E, I) is the disjoint union of two graphs {Ei, Ii) and {E2, 12), then 
G{E, I) = G{Ei, h) * G(i:2, 12). If {E, I) is obtained from (^i, 7i) and {E2, h) by 
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Fig. 2. The graphs C4 and P4 



connecting each element of Ei to each element of E2, then I) = x 
G{S2, h) - Graph groups were studied e.g. in [13]; they are also known as free partially 
commutative groups [10,40], right-angled Artin groups |6,7], and semifree groups [2]. 

A transitive forest is an independence alphabet {S, I) such that there exists a forest 
F of rooted trees (i.e., a disjoint union of rooted trees) with node set S and such that 
for all a,b G E with a ^ b: (a, b) e 7 if and only if a and b are comparable in F 
(i.e., either a is a proper descendant of 6 or & is a proper descendant of a). It can be 
shown that {E, I) is a transitive forest if and only if {E, I) does not contain an induced 
subgraph, which is a cycle on 4 nodes (also called C4, see Figure 2 on the left) or a 
simple path on 4 nodes (also called P4, see Figure 2 on the right) [39]. The next lemma 
follows easily by induction. We sketch the proof. 

Lemma 2. The class C of all groups, which are of the form G^E, I) for a transitive 

forest {E, I), is the smallest class such that: 

(1) l€C 

(2) ifGi,G2 e C, then also Gi * G2 G C 

(3) ifG GCthenGxZeC 

Proof. First we verify that graphs groups associated to transitive forests satisfy (l)-(3). 
Case (1) results from the empty graph. It is immediate that transitive forests are closed 
under disjoint union, which implies (2). If F is a forest of rooted trees, then one can 
obtain a rooted tree by adding a new root whose children are the roots of the trees from 
F. On the group level this corresponds to (3). 

For the converse, we proceed by induction on the number of vertices. If the forest 
{E, I) consists of more than one rooted tree, then G{E, I) is the free product of the 
graph groups associated to the various rooted trees in {E,I), all of which have a smaller 
number of vertices. If there is a single tree, then in {E, I) the root is connected to every 
other vertex. Thus G{E, I) = G x Z where G is the graph group corresponding to the 
transitive forest obtained by removing the vertex corresponding to the root and making 
its children the roots of the trees in the forest so obtained. □ 

Of course, a similar statement is true for trace monoids of the form M.{E,I) with 
{E, I) a transitive forest; one just has to replace in (3) the group Z by the monoid N. 

3 The rational subset membership problem 

Let C be the smallest class of groups such that: 
- the trivial group 1 belongs to C 
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- if G e C and H < Gis finitely generated, then also H eC 

- if G G C and G < H such that G has finite index in H (i.e., H isa finite extension 
of G), then also i? e C 

- if G e C, then also G x Z e C 

- if A is a finite graph of groups [36] whose edge groups are finite and whose vertex 
groups belong to C, then the fundamental group of A belongs to C (in particular, 
the class C is closed under free products). 

This last property is equivalent to saying that C is closed under taking amalgamated 
products over finite groups and HNN-extensions with finite associated subgroups [36]. 
The main result in this section is: 

Theorem 1. For every group G €C, the rational subset membership problem is decid- 
able. 

It is well known that decidability of the rational subset membership problem is pre- 
served under taking finitely generated subgroups and finite extensions [20]. Moreover, 
the decidability of the rational subset membership problem is preserved by graph of 
group constructions with finite edge groups [24]. Hence, in order to prove Theorem 1, 
it would suffice to show that the decidability of the rational subset membership problem 
is preserved under direct products by Z. But currently we can neither prove nor disprove 
this. This forces us to adopt an alternate strategy: we will introduce an abstract property 
of groups that implies the decidability of the rational subset membership problem, and 
which has the desired closure properties. 

Let £ be a class of formal languages closed under inverse homomorphism. A finitely 
generated group G is said to be an C-group if WP i;(G) belongs to C for some finite 
generating set E. This notion is independent of the choice of generating set [17, 21, 24]. 

A language Lg ^ ^* belongs to the class RID (rational intersection decidable) 
if there is an algorithm that, given a finite automaton over S recognizing a rational 
language L, can determine whether Lq C\ L ^ was shown in [24] that the class 
RID is closed under inverse homomorphism and that a group G has a decidable rational 
subset membership problem if and only if it is an RlD-group. This follows from the fact 
that if i is a rational subset of a group G, then e L if and only if 1 e g~^L and that 
g~^L is again a rational subset. 

Let C 0* be a language over an alphabet 0. Then K belongs to the class SLI 
(semilinear intersection) if, for every finite alphabet F (disjoint from 0) and every 
rational language L C. (6> U T)*, the set 

^r{{w e L I TTe{w) e K}) = ^riL D ttq\K)) (2) 

is semihnear, and the tuples in a semihnear representation of this set can be effectively 
computed from F and a finite automaton for L. This latter effectiveness statement will 
be always satisfied throughout the paper, and we shall not exphcitly check it. In words, 
the set (2) is obtained by first taking those words from L that project into K when F- 
letters are erased, and then erasing the ^-letters, followed by taking the Parikh image. 

In a moment, we shall see that the class SLI is closed under inverse homomorphism, 
hence the class of SLI-groups is well defined. In fact, we show more generally that the 
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class SLI is closed under inverse images by sequential functions [3]. This will imply, 
moreover, that the class of SLI-groups is closed under taking finite extensions [17, 21, 
24]. 

A sequential transducer A with input alphabet S and output alphabet J? can be 
defined as a finite state automaton with transitions labeled by elements from the set 
E X Q* such that the following restriction is satisfied: If there are states q, qi , (?2 and a 
transition from q to qi (for i e {1, 2}) with label (a, Wi) £ S x then qi ~ q2 and 
Wi = W2 - This is not the standard definition of a sequential transducer (see e.g. [3]), but 
it is easily seen to be equivalent. The language defined by A is a relation RC S* xQ*, 
and it is easy to see that R is the graph of a partial function f : S* ^ fi*.A sequential 
function is a partial function, which is computed by a sequential transducer. 

Lemma 3. Let K C 0* belong to SLI and let f : S* ^ 0* be a sequential function. 
Then f~^{K) belongs to SLL In particular, the class of SLI-groups is well defined and 

is closed under taking finite extensions. 

Proof. Let T be an alphabet disjoint from S and let i be a rational subset of U Z')*. 
Let ^ be a sequential transducer computing the sequential function f : E* ^ Q*. 
Define a transducer A' by adding to each state of A a loop with label (a. a) for each 
a € r. Clearly, A' is a sequential transducer, which computes a sequential function 

F:{ruE)*^ (ru6))*. 

The following two observations are immediate from the fact that the only transitions 
of A' involving letters from F are loops with labels of the form (a, a): 

(a) coincides with <Fr on the domain of F (we read the composition of functions 
from right to left, i.e., in \I'rF we first apply F, followed by >f'r) 

(b) TTeF = f-Ks. 

We now claim that the following equality holds: 

F{L n iT-^\f-\K))) = F{L) n -.^^K). (3) 

First note that L n TT^^{f-^{K)) = L n F-^{t^q^{K)) by (b). So if w belongs to 
the left hand side of (3), then w = F{u) with u e L n F-'^{tt0^{K)). Thus w e 
F{L) n tTq^ (JsT). Conversely, if u £ F{L) n tt^^'^{K), then there exists w £ L such that 
= u. But then w e L n F-^{ttq\K)) = Ln n'^^{f-^{K)) and so u belongs 
to the left hand side of (3). 

Now, since L n tt'^^ {f^^{K)) = Ln F~^(7r@^(ii')) is contained in the domain of 
F, we may conclude from (a) and (3) that 

^r{L n 7r^\f-\K))) = ^rF{L n 7:i\f-\K))) = ^r{F{L) n t^q^K)). (4) 

But F{L) is rational since the class of rational languages is closed under images via 
sequential functions [3]. Therefore, since K belongs to SLI, we may deduce that the 
Parikh-image '^r{F{L) n tt^,^ (if )) is semilinear. This completes the proof of the first 
statement from the lemma in light on (4). 

Since a homomorphism is a sequential function, the language class SLI is closed 
under inverse homomorphism. Hence, the class of SLI-groups is well defined. Finally, 
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let us assume that G is an SLI-group and that G is a finite index subgroup of H. Let 
E (resp. A) be a finite generating set for G (resp. H). Then in [24, Lemma 3.3] it is 
shown that there exists a sequential function f : A* E* such that WP^i (H) = 
f-\WPs{G)). Hence, F is an SLI-group. □ 

Let us quickly dispense with the decidability of the rational subset membership 
problem for SLI-groups. 

Lemma 4. The class of languages SLI is contained in the class of languages RID. In 
particular, every SLI-group has a decidable rational subset membership problem. 

Proof. Let K 0* belong to SLI. Let A be a finite automaton over the alphabet &. 
We have to decide whether L(A) C\K ^ ^. Since K belongs to SLI, the set 

^^{{w e L{A) I Tieiw) G K}) = ML{A) n K) 

is effectively semihnear and so has a decidable membership problem (c.f. [24]). As 
mentioned earUer, d'0(L(A) n K) consists of the unique function — > N if L{A) n K 
is non-empty and is empty otherwise. Thus we can test emptiness for L{A) OK. □ 

Having already taken care of finite extensions by Lemma 3, let's turn to finitely 
generated subgroups. We show that the language class SLI is closed under intersection 
with rational subsets. This guarantees that the class of SLI-groups is closed under taking 
finitely generated subgroups [21]. 

Lemma 5. Let K CO* belong to SLI and let R C- 0* be rational. Then RnK belongs 
to SLI. In particular, every finitely generated subgroup of an SLI-group is an SLI-group. 

Proof. Let L C (r U 0)* be rational, where T is a finite alphabet disjoint from 0. We 
have 

Lmr0\RnK) = Lmr0\R)n'K0\K). 

But rational languages are closed under inverse homomorphism and intersection, so 
^r{L n 7r@^(i?) n 7r@^(-fC)) is semilinear as K belongs to SLI. This establishes the 
lemma. □ 

Next, we show that the class of SLI-groups is closed under direct products with Z: 

Lemma 6.IfG is an SLI-group, then G x Zis also an SLI-group. 

Proof. Let i7 be a finite generating set for G. Choose a generator a ^ 17 of Z. Then 

G X Z is generated by T U {a}. Let be a finite alphabet (F n (S"^^ U {a, a~^}) = 0) 
and let L be a rational subset of (S^^ U {a, ar^} U F)*. We have 

^r({w € L I 7r^±iu{a,a-i}H e WP£u{a}(G X Z)}) = 

Tfr(^'^ru{a,a-^}{{w G L I 7rj:±i(u;) e WPi;(G)}) n 

This set is semihnear, since {/ e N^u{a,a 1} I ^ /(a-i)}is semilinear and 
semilinear sets are closed under intersection and projection [18]. □ 
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By Lemmas 3-6, Theorem 1 would be established, if we could prove the closure 
of C under graph of groups constructions with finite edge groups. Unfortunately we are 
only able to prove this closure under the restriction that every vertex group of the graph 
of groups is residuaUy finite (which is the case for groups in C). In general we can just 
prove closure under free product. This, in fact, constitutes the most difficult part of the 
proof of Theorem 1 . 

Lemma 7. IfGi and G2 are SLI-groups, then G\ * G2 is also an SLI-gwup. 

Proof. Assume that Ei is a finite generating set for Gi. Thus, E = Si U IJ2 is a 
generating set for the free product Gi * G2. Let _r be a finite alphabet (F n = 0) 
and let = S^^ U F. Let L C 0* be rational and let A = {Q, O, S, qq, F) be a finite 
automaton with L = L{A), where Q is the set of states, S C Q x O x Q isthe transition 
relation, qa G Q is the initial state, and F C Q is the set of final states. For p,q G Q 
and w e 6** we write p q if there exists a path in A from p to q, labelled by the 
word w. 

For every pair of states {p,q) & Q x Q let us define the language 

L\p, q] C U r U (Q X Q))* U {Et^ U T U (Q x Q))* C (0 u (Q x Q))* 
as follows: 

L\p,q]= U {'wo{pi,qi)wi{P2,q2)---wk-i{Pk,qk)'Wk\ 

k>l A (pi,gi),...,(pfe,gfc) e Q X Q A 

wo,...,wke {Ef^ u ry A TT^ii {wo ■■■wk)e m>s, (Gi) a 

Wo , tUl A A Wh-1 A "'fc 1 

P — >A Pi A qi — >A P2 A • • • A qk-i >a Pk A qk — >a q} 

Since the language 

{woipi,qi)wi{p2,q2) ■ ■ ■Wk-iipk,qk)wk \ 

k>l A {puqi),...,{pk,qk) eQxQ A wo,...,Wke {S^^ U F)* A 

Wo Wi A A Wk^l A 1 

P — >A Pi A qi — >A P2 A • • • A qk-i >a Pk A qk — >a q} 

is a rational language over the alphabet Sf^ U T U (Q x Q) for i G {1, 2} and Gi 
is an SLI-group, it follows that the Parikh image llfruiQxQ^iLlp, q]) C f^^^iQxQ) is 
semilinear. Let K\p, q] C (F \J {Q x Q))* he some rational language such that 

1r{K[p,q])^1rruiQ>cQ){L[p,q]). (5) 

Next, we define a context-free granmiar G = (A'^, F, S, P) as follows: 

- the set of nonterminals is A'' = {S} U {Q x Q), where 6" is a new symbol not 
contained inQ x Q. 

- 5 is the start nonterminal. 
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- P consists of the following productions: 

S {qo,qf) for all g/ G F 
{p, q) K\p, q] for all p,q&Q 
(?) q) ^ ^ for all g G Q 

By Parikh's theorem, the Parikh image !Z'(L(G)) C is semilinear. Thus, the follow- 
ing claim proves the lemma: 

Claim 1. ^iL{G)) = ^ri{w e L{A) \ 'Kj:±i{w) e WPi;(Gi * G2)}) 

Proof of Claim 1. We prove the following more general identity for all {p, q) £ Q x Q: 

^{L{G, (p, q)))=^r{{w€e* \p^Aq A 7r^±i H € WPi:(Gi * G2)}) 
For the inclusion 

(L(G, {p, q))) C^r{{wG0* \p^Aq A Tr^ii (w) € WPs{Gi * G2)}) (6) 

assume that (p, q) =>g u £ F*. We show by induction on the length of the G-derivation 
(P)<?) ti that there exists a word w € 0* suchthatp q,'r7s±i{w) € WPi;(Gi* 
G2), and ^{u) = iPriw). 
Case 1. p = q and u = s: We can choose w = s. 

Case 2. {p,q) u' =>q u for some u' £ K[p,q]. By (5), there exists a word 
V e F[p,q] such that 'F{u') — 'Fru{QxQ){v). Since G -^^[p, 9], there exist A; > 1, 
{pi,qi),...,{pk,qk) &QxQ,ie {l,2},andt;o,---,t^fe € (rf ^ U T)* such that 

- p — >A Pi, qi — >A P2, • • • , Pk, qk — >A 

- = t;o(pi, gi)t;i(p2, ^2) • • • Vk-i{pk, qk)vk, and 

- TT^ii (t;o ■ • • Wfe) e WPi;, {Gi). 

Since u' =4>g u G F* and = !Z'ru(QxQ)(^)> there must exist ui,...,Uk € .T* 

such that 

iPi,qi)^GUi and tf'(u) = <Z'r(wo) + ■ • • + ^'r(wfe) + '^'(wi) + ■ • • + 

for all 1 < i < fc. By induction, we obtain words wi,. . . ,Wk € 0* such that for all 

1 < j < A;: 

Wi 

- Pi — >A qi 

- TTs±i{wi) G WPi;(Gi * G2), and 

- ^{ui) = ^r{wi). 

Let us set w = vqWiVi ■ ■ ■ WkVk G 0*. We have: 

Vo Wl Vl Wk D|t . to 

- p — >A Pi — ^A qi — >A P2 ■ ■ -Pk — 'A qk — 'A q, i.e., p -^a q, 

- TTs±i (w) e WPsiGi * G2), and 

- ^{u) = i^r{vo) + - ■ ■+^r{vk)+1'{ui) + - ■ ■+^{uk) = ^r{vo) + - ■ ■ + ^r{vk) + 
^r{wi) H h ^r{wk) = ^r{w). 
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This concludes the proof of inclusion (6). For the other inclusion, assume that 

p^AQ and 7ri;±i(w) e WPi;(Gi * G2) 
for a word w £ 6>*. By induction over the length of the word w we show that ^r{w) & 

We will make a case distinction according to the three cases in Lemma 1 . Note that 
we either have w € -T* or the word w € 0* can be (not necessarily uniquely) written 
asw = wi---Wn with n > 1 such that Wi e {{r U S^^)* U (r U X"^^)*) \ r* and 

WiG{ru st^)* ^ Wi+i G (r u sf^)*. 

Case 7. w e (r U E^^)* (the case w G {F U E^^)* is analogous): Then tt^±i (w) G 

WP^j (Gi). Together with ]3 g, we obtain ^(g, = w{q,q)s G g]. Since 
(P; q) ^ K\p, q] and (g, g) ^ e are productions of G, there exists a word u £ F* such 
that (p, g) =^G u and = ^'rC^'), i.e., !i'r(w) G !i'(-£'(G, (p, q))). 

Case 2. w = W1W2 with wi 7^ e 7^ W2 and TTx'ii Tr^'ii («^2) G WPi;(Gi * G2). 
Then there exists a state r € Q such that 

P — »A r — >A q- 

By induction, we obtain 

^r{wi) G !p-(L(G, (p,r))) and 
!Z^r(«^2)e-i^(L(G,(r,g))). 

Hence, we get 

^r{w) = ^riwi) + <Z'r(w2) 

G!Z^(i(G,(p,r)))+!Z^(L(G,(r,g))) 
C!i^(L(G,(p,g))), 

where the last inclusion holds, since {p,r){r,q) G L[p,q], and so either {p,q) — > 
(p, r){r, q) or (p, g) — > (r, g)(p, r) is a production of G. 

Case 3. w — vqWiVi ■ ■ ■ WkVk such that fc > 1, 

- TT^ii {wi) G WPi;(Gi * G2) for alH G {1, . . . , k}, and 

- for some i G {1,2}: vo,...,Vk G {F U E^^)* \ F* and tt^±i{vo ■ ■ -Vk) G 
WPi:,(Gi). 

There exist states Pi,qi, ■ ■ ■ ,Pk,(lk S Q such that 

1)0 Wl Vl MJfe IJfe 

p — >A Pi — >A qi — >A P2 • • • Pfe — >A gfe — >A g- 

By induction, we obtain 

^riwi)G^{L{G,{pi,qi))) (7) 
for all 1 < i < fc. Moreover, from the definition of the language L\p, q] we obtain 

V = vo{pi,qi)vi{p2, 52) • • • Vk-i{pk, qk)vk e L\p, q]. 
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Hence, by (5) there is a word u' G K\p,q] such that \I/{u') = !Z'ru(QxQ)(^) a^id 
(p> q) ^ w' is a production of G. With (7) we obtain 

for a word u ^ F* such that 

<f (u) = ^r{vo) + ■■■ + ^r{vk) + ^r{wi) + ■■■ + ^r{wk) = ^r{w), 

i.e., Hj^riw) S ^{L{G, {p, q))). This concludes the proof of Claim 1 and hence of the 
lemma. □ 

If we were to weaken the definition of the class C by only requiring closure under 
free products instead of closure under finite graphs of groups with finite edge groups, 
then Lemmas 4-7 would already imply Theorem 1 . In fact, this weaker result suffices 
in order to deal with graph groups, and readers only interested in graph groups can skip 
the following considerations concerning graphs of groups. 

To obtain the more general closure result for the class C concerning graph of group 
constructions, we reduce to the case of free products. Recall that a group G is residually 
finite if, for each g G G \ {1}, there is a finite index normal subgroup N of G with 
g ^ N. Now we use a standard trick for graphs of residually finite groups with finite 
edge groups. 

Lemma 8. Let A be a finite graph of groups such that the vertex groups are residually 
finite SLI-groups and the edge groups are finite. Then the fundamental group of A is an 
SLI-group. 

Proof. Let G be the fundamental group of A. Then G is residually finite [36, 11.2.6 
Proposition 12]. Since there are only finitely many edge groups and each edge group 
is finite, there is a finite index normal subgroup N < G intersecting trivially each 
edge group, and hence each conjugate of an edge group. Thus the finitely generated 
subgroup N < G acts on the Bass-Serre tree for G [36] with trivial edge stabilizers, 
forcing to be a free product of conjugates of subgroups of the vertex groups of G and 
a free group [36]. Since N is finitely generated, these free factors must also be finitely 
generated. Since every finitely generated subgroup of an SLI-group is an SLI-group 
(Lemma 5) and Z is an SLI-group (Lemma 6), we may deduce that is a free product 
of SLI-groups and hence is an SLI-group by Lemma 7. Since G contains AT as a finite 
index subgroup. Lemma 3 implies that G is an SLI-group, as required. □ 

Clearly, the trivial group 1 is an SLI-group. Also all the defining properties of C 
preserve residual finiteness (the only non-trivial case being the graph of group con- 
structions [36]). Hence, Lemmas 4-6 and Lemma 8 innmediately yield Theorem 1. 

Our main apphcation of Theorem 1 concerns graph groups: 

Theorem 2. The rational subset membership problem for a graph group /) is 
decidahle if and only if{S, I) is a transitive forest. Moreover, if{S, I) is not a transitive 
forest, then there exists a fixed rational subset L of<G(S, I) such that the membership 
problem for L within G{S, I) is undecidable. 



16 



Proof. The decidability part follows immediately from Theorem 1: Lemma 2 imphes 
that every graph group G(i7, 1) with {S, I) a transitive forest belongs to the class C. 

Now assume that {S, I) is not a transitive forest. By [39] it suffices to consider the 
case that [S, I) is either a C4 or a P4. For the case of a C4 we can use Mihailova's 
result [30]. Now assume that {E, I) is a P4. We will reuse a construction by Aalbers- 
berg and Hoogeboom [ 1 ] , which is based on 2-counter machines. A 2-counter machine 
is a tuple C = {Q, Ins, q^, qf) where Q is a finite set of states, qQ G Q is the initial 
state, qj G Q is the final state, and Ins C Q x {il,i2,dl,d2, zl, z2,pl,p2} x Q is 
the set of instructions. The set of configurations of C is Q x N x N. For two configura- 
tions {q, m, ^2), {q\ nil, 1712) we write {q, rii, 712) =^c [q' , mi, 1712) if there exists an 
instruction {q,ak,q') G Ins, so a G {i,d, z,p},k G {1,2}, such that ms-k = ns-fe 
and one of the following three cases holds: 

- a = i and = + 1 

- a = d and ruk = n/fc — 1 

- a = z and uik = = 

- a = p and ruk = > 

Since Turing machines can be simulated by 2-counter machines [29], it is undecidable 
whether for a given 2-counter machine C — (Q, Ins, go, 9/) there exist m, n G N 
with (go, 0, 0) (g/, m, n). In [1], this problem is reduced to the question, whether 
Lr\K = 9for given rational trace languages L,K C M{S, I), where E = {a, b, c, d} 
and / = {{a,b), (6, c), (c, d)}. In fact, the language K is fixed, more precisely 

K = ba{d{cb)+a)*dc* 

= {[ab>''c'^dab>'c'^d---ab''-'c''d]i \ I > 1, jo = 1, ji, • • -Jt > !}• 

The problem is that in the construction of [1] the language L is not fixed since it 
depends on the 2-counter machine C. Aalbersberg and Hoogeboom encode the pair 
of counter values (m, n) G N x N by the single number 2™3". The language L 
is constructed in such a way that K n L contains exactly those traces of the form 
[aV°c^^dab^^c^''d- ■ ■aV<'-^c^'d]i, such that i > I, jo = 1, and there exist states 
qi,...,qe and mi,ni, . . .,mi,ne G N with g^ = Qf, 2™' 3"* = ju and (go, 0,0) 
(gi, 777,1 . ni) (92, ™2, '^2) =>c ■■■ =^c {li^iT^i^'nt) (note that jo = lencodesthe 
initial counter values (0,0)). 

In order to construct a fixed rational subset of G(i7, 7) with an undecidable member- 
ship problem, we start with a fixed (universal) 2-counter machine C = {Q, Ins, go, g/) 
such that it is undecidable whether 3777', 77' G N : (go, to, 77) =^>^ (g/, to', 77') for given 
natural numbers m, n. Such a machine C can be obtained by simulating a universal 
Turing machine. Let L C M.{E, I) be the fixed rational trace language constructed 
by Aalbersberg and Hoogeboom from C, and let us replace the fixed trace language 
K = ba{d{cb)~^ a)* dc* by the (non-fixed) language 

Km,n = b'^'"^"a{d{cb)+a)*dc* 

= {[ab>°c:''dab>'d>^d---ab>'-'c:''d]i \ £ > 1, jo = 2™3", ji, . . .,je > 1}. 

Then it is undecidable, whether Km.n H L ^ for given 777, 77 G N. Hence, it is unde- 
cidable, whether 6"^"^" G a{d{cb)~^a)*dc*L~^ in the graph group G{E, I). Clearly, 
a{d{cb)~^a)*dc*L~^ is a fixed rational subset of the graph group G{S, I). □ 
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We conclude this section with a further application of Theorem 1 to graph products 
(which should not be confused with graphs of groups). A graph product is given by a 
tuple {S, I, (Gu)wei:). where {S, I) is an independence alphabet and Gy is a group, 
which is associated with the node v G S. The group I, (G„)„gi;) defined by this 
tuple is the quotient 

G(i:,7,(G„)„ei:) = * Gy/{xy = yx\xeGu,yGGy,{u,v)eI}, 

i.e., we take the free product G„ of the groups G„ (v G S), but let elements from 
adjacent groups commute. Note that G{S, I, (G^)„gj:) is the graph group G{S, I) in 
the case every Gy is isomorphic to Z. Graph products were first studied by Green [19]. 

Theorems. If is a transitive forest and every group Gy (v € V) is finitely 

generated and virtually Abelian ( i.e., has an Abelian subgroup of finite index), then the 
rational subset membership problem for G{S, /, {Gy)y(zs) is decidable. 

Proof. Assume that the assumptions from the theorem are satisfied. We show that 
G(i7, /, (Gt,)t,gs) belongs to the class C. Since (U, /) is a transitive forest, the group 
0(17, /, {Gy)yi^s) can be built up from trivial groups using the following two opera- 
tions: (i) free products and (ii) direct products with finitely generated virtually Abelian 
groups. Since the class C is closed under free products, it suffices to prove that if G 
belongs to the class C and H is finitely generated virtually Abelian, then G x H also 
belongs to the class C. As a finitely generated virtually Abelian group, H is a finite 
extension of a finite rank free Abelian group Z". By the closure of the class C under 
direct products with Z, G x Z" belongs to the class C. Now, G x is a finite extension 
of G X Z", proving the theorem, since C is closed under finite extensions. □ 

4 The submonoid membership problem 

Recall that the submonoid membership problem for a group G asks whether a given 
element of G belongs to a given finitely generated submonoid of G. Hence, there is a 
trivial reduction from the submonoid membership problem for G to the rational subset 
membership problem for G. We will show that for every amalgamated free product 
G *A H such that: 

1. A = G n H is a finite, proper subgroup of G and H; 

2. there exist g e G, h e H with g~^Ag n A = 1 = h~'^Ah n A, 

there is in fact also a reduction in the opposite direction. Similarly, if >l<^ G is an HNN 
extension with : A ^ B with 

1 . A is a finite subgroup of G; 

2. there exists g eG such that g~^Ag n A = 1 or g~'^Ag n S = 1 

then the rational subset problem reduces to the submonoid membership problem for 

*^ G. We remark that in 2, one could by symmetry switch the roles of B and A. 

Using the following lemma, it will suffice to consider a free product G * F^, where 
F2 is a free group of rank two. 
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Lemma 9. Let G* a H be an amalgamated free product such that H ^ A,[G : A] > 5, 
and there exists h E H with h^^Ah riA=l. Then G*aH contains as a subgroup the 
free product G * F2 of G with a free group of rank two. 

Proof Since [G : A\ > 5, we can choose elements 31,52, 93, 54 € G\A which 
belong to pairwise distinct left A-cosets. Moreover, choose an element h £ H\A with 
h^^Ah n A — 1. First we claim that x ~ gxhg^^ and y = g^hg^"^ freely generate a 
free subgroup of G *a H. For this, note that g~^gj €G\Aifi^j. Thus, every word 
over {x, x~^ ,y,y~^} which does not contain a factor from {xx~^,x~^x, yy~^ ,y~^y} 
yields a reduced sequence for the amalgamated product. The normal form theorem for 
amalgamated free products [26, Chapter IV, Theorem 2.6] then implies that {x, y} is the 
base of a free subgroup of G *a H. Hence, the conjugates u = hxh~^ and v = hyh~^ 
also form a base for a free subgroup of G *a H. Since h~^Ah D A = 1 (and hence 
a a € A, then h~^ah e H \ A) a word over G \ {1} U {u,u~^ ,v,v~^}, which 
does not contain a factor from {G \ {1}) {G Li {uu~^ ,u~^u,vv~^ ,v~^v}, yields 
a reduced sequence for the amalgamated product. Again, the normal form theorem for 
amalgamated free products imphes that the subgroup oiG*AH generated by G[J{u,v} 
is isomorphic to G * □ 

We now prove the analogous result for HNN extensions. 

Lemma 10. Let ^^G be an HNN extension with stable letter t and finite associated 

subgroups A, B (so if : A B) such that [G : B] > 3 and there exists g G G with 
g~^AgnA = 1 or g'^AgOB = 1. Then 4^ ^ G contains as a subgroup the free product 
G * F2ofG with a free group of rank two. 

Proof. By Lemma 9, it suffices to show that G contains a subgroup G * Z. We may 
assume that A 1 B, because otherwise *^ G ~ G * Z. Choose ,gi, 52 E G \ B 
so that gi , 52 are in different left cosets of B. Suppose first there exists g & G with 
g~^Ag n ^ = 1 and set x = git~^gtg2^. Since g ^ A (because otherwise A = 1) and 
92^91 ^ B, one easily deduces that x" is a reduced sequence for the HNN extension for 
all n > and hence x is of infinite order by Britton's lemma. Set y = t~^gtxt~^g~^t. 
Then y is of infinite order, being a conjugate of x. We claim that G and (y) generate 
their free product inside of *y G. We need to show that a word over G\{1}L) {y.y^^} 
with no factor from {G\{l}){G\{l})U{yy~^ ,y~^y} results in a reduced sequence for 
the HNN extension. The key point is that if h G G\B, then g~^tht~^ gt is reduced. 
On the other hand, if 6 e B \ {1}, then t''^ g-HU-'^ gt = t-'^g-'^ip-'^{h)gt-'^, which 
is reduced since g~^Ag n ^ = 1. 

Now assume that there exists g & G with g~^Ag f) B = 1. The group A must be 
a proper subgroup of G, because otherwise we have 1 = g~^Ag r\B = GriB = B. 
So choose go € G\A and set x = git~^gotg2^- The same argument as above shows 
that X has infinite order. Set y = t~^gt~^xtg~^t; again y has infinite order, being a 
conjugate of x. Again, we claim that G and (y) generate their free product in G. 
Once more, we must prove that a word over G \ {1} U {y, y~^} with no factor from 
(G \ {1})(G \ {1}) U {yy~^, y~^y} yields a reduced sequence for the HNN extension. 
The key point is that if /i G G \ B, then tg~^tht~^gt~^ is reduced. On the other 
hand, if 6 € -B \ {1}, then tg~^tbt~^gt~^ = tg~^(p~^{b)gt~^, which is reduced since 
g~'^Agr\B = 1. □ 
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The following lemma is cracial for us: 
Lemma 11. 

(1) Let G and H be finitely generated groups such that the finite group A is a proper 
subgroup of both G and H and there exists h G H with h~^Ah Ci A = 1. Then 
the rational subset membership problem for G can be reduced to the submonoid 
membership problem for G *a H. 

(2) If (f : A ^ B is an isomorphism between finite subgroups of a finitely generated 
group G and there exists g G G with g^^Ag D A = 1 or g~^Ag 8 = 1, then 
the rational subset membership problem for G can be reduced to the submonoid 
membership problem for ^^G. 

Remark 1. In our proof of Lemma 11 we will implicitly construct Turing machines that 

carry out the reductions in (1) and (2). These machines will depend on the element g 
(and /i) in (1), respectively (2). Here one might argue that these elements are not known. 
But this is not a real problem, since g and h are fixed elements which do not depend on 
the input for the reduction. So there exists a Turing machine that can do the reduction, 
although we don't know which Turing machine if we don't know the elements g and h. 

Proof of Lemma IL If G is finite, then the rational subset membership problem for G 
is decidable, so we may assume without loss of generality that G is infinite. Since A 
is finite, we have [G : A] > 5 in (1), respectively [G : B] > 3 in (2). Then Lem- 
mas 9 and 10 imply that G * F2 is a subgroup of G *a H, respectively G. Since 
the submonoid membership problem for a finitely generated subgroup of a group K 
reduces to the submonoid membership problem for K itself, it suffices to prove the 
following: the rational subset membership problem for G can be reduced to the sub- 
monoid membership problem for G * F2. Let 17 be a finite generating set for G and use 
h : (r^i U r^Y G * for the canonical morphism. Let A = {Q, (5, q^, F) 
be a finite automaton and let t £ (iJ^^)*. By introducing e-transitions, we may assume 
that the set of final states F consists of a single state <// ^ go- One can effectively find 
a subset Q C F2 in bijection with Q via q ^ q such that Q freely generates a free 
subgroup of F2. 

We construct a finite subset C {S^^\jr^^)* and an element u e (iJ^^ur^i)* 
such that h{t) G h{L{A)) if and only if h{u) G h{A*). Let 

A = {qcp~^ \ {q,c,p) & 5} and u = qotqJ^. (8) 

Note that in (8), we have c G U {e}, since we introduced £-transitions. Recall 
{q, c,p) G 5 means q pin A. We begin with a critical claim. 

Claim 1. Suppose that in G * F2, we have 

qotqj'^ = (Pii'igf^) • • • (PnVnqn^) (9) 
where pi ^ qi in A, for i <E {1, . . . , n}. Then h{t) e h{L{A)). 

The claim is proved by induction on n. If n = 1, then since 7^ 9/> the normal form 
theorem for free products easily implies qo = pi, qf = qi and t = vi in G. Thus 
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Qo Qf in A, whence vi G L{A), and so h{t) G h{L{A)). Next suppose the claim 
holds forn — 1 > 1 and consider the claim forn > 1. 
First suppose that qi = for some i. Then 

qotqj'^ = (piwigf^) • • • (PiViVi+iq^^i) ■ ■ ■ {PnVnqn^) 

in G * F2 and pi "'"'"""^ qi+i in A. Induction now gives the desired conclusion. 
Next suppose that for some i, we have pi = qi and Vi = linG. Then 

qo t qj'^ = {piviq^'^) ■ ■ ■ {pi-iVi-iq^\){pi+iVi+iq^_^^) ■ ■ ■ {PnVnq~^) 

inG * F2 and we can again apply the induction hypothesis. 

Finally, suppose pi = qi implies Vi 7^ 1 in G and suppose qi ^ Pi+i, all i. Then 
we claim that the right hand side of (9) is already in normal form. Consider a typical 
window ql^iPiViq~^Pi+i (where we take qo = 1 = Pn+i)- Then no two neighbouring 
elements belong to the same factor of the free product G * {Q) = G * (si) * • • • * 
where Q = {si, . . . , s„}, since qj ^ pj+i for j = i — 1, z and Pi ^ qi when Vi = 1 
in G. Since such windows cover the right hand side of (9) we may conclude that it is 
in normal form in G * i^2- Comparison with the left hand side then shows that n = 1, 
contradicting n > 1. So this case does not arise and the proof of the claim is complete. 

Now we may prove that h{t) e h{L{A)) if and only if h{u) G h{ A*). Suppose first 
that h{t) = h{t') with t' e L{A). Write t' = af-a„ with e S^'^ U {e} and such 
that qo qi 92 — > ■ • • — > 9n-i If - Then, as h{t) = h{t'), clearly we 
have 

u = qot'qf~^ = {qoaiqi^){qia2q2^) ■ ■ ■ {qn-ianqj^) e A* 
in G * -F2- Conversely, suppose h{u) e h{A*). Then we can write 

u = qotqj'^ = (piaigf^) ■ • • (pnanqn^) 

in G * F2. where Pi qi are certain transitions of A. Claim 1 then implies h{t) G 
h{L{A)). □ 

Theorem 4. Let G and H be finitely generated groups such that the finite group A is 
a proper subgroup of both G and H and there exist g E G, h E H with g^^Ag n 
A = 1 = h~^Ah n A. Then, for the amalgamated free product G *a H the rational 
subset membership problem and the submonoid membership problem are recursively 
equivalent. 

Proof It suffices to show that the rational subset membership problem for G*aH can 
be reduced to the submonoid membership problem for G *a H. The rational subset 
membership problem for G *a H can be reduced to the rational subset membership 
problems for G and H [24]. By Lemma 1 1 both these problems can be reduced to the 
submonoid membership problem for G *a H. □ 

Note that the assumptions in Theorem 4 are satisfied for every free product G * H 
of nontrivial finitely generated groups G and H. 
A similar result holds for HNN extensions: 
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Theorem 5. Let G be a finitely generated group and let (p : A ^ B be an isomorphism 
between finite subgroups of G. Suppose there exists g E G, with g^^Ag D A = 1 or 
g~^Ag n B = 1. Then the rational subset membership problem and the submonoid 
membership problem are recursively equivalent for the HNN extension ^^G. 

Proof. We just need to establish that the rational subset membership problem for *^ G 
can be reduced to the submonoid membership problem. The rational subset membership 
problem for ^^pG can be reduced to the rational subset membership problem for G 
by the results of [24]. By Lemma 11 this problem can be reduced to the submonoid 
membership problem for *^ G. This completes the proof. □ 

Let us say that a group G is virtually a free product if it has a finite index subgroup 
H that splits nontrivially as a free product H = Gi * G2. 

Corollary 1. Let G be a finitely generated group that is virtually a free product. Then 
the rational subset and submonoid membership problems are recursively equivalent. 

Proof. Suppose G has decidable submonoid membership problem. We need to show 
that G has decidable rational subset problem. Let i7 be a finite index subgroup of G that 
splits nontrivially as a free product. Clearly H has decidable submonoid membership 
problem and hence has decidable rational subset membership problem by Theorem 4. 
It then follows G has decidable rational subset membership problem by [20, 24]. □ 

In order for a finitely generated group to be virtually a free product, it must have two 
or more ends. On the other hand, a group with two or more ends that is either virtually 
torsion-free or residually finite is easily seen, via StaUings ends theorem [38], to be 
virtually a free product, as we now show. First we recall the notion of ends of a locally 
finite graph. 

Let rhea locally finite graph, i.e., every node of F has only finitely many neigh- 
bours. Consider the inverse system F \ C where G runs over the finite subgraphs of 
F. Then the sets of connected components ttq {F \ C) form an inverse system of sets; 
the projective Umit Ends(/^) = lim7ro(-r \ C) is known as the set of ends of F. The 
number of ends of F is the cardinality of Ends(F). The number of ends of a finitely 
generated group G is the number of ends of the Cayley-graph of G with respect to 
any finite set of generators; this number is independent of the finite generating set we 
choose for G and it is either 0, 1, 2 or 00 [38]. Here are some examples: (i) every finite 
group has ends, (ii) Z x Z has one end, (iii) Z has two ends, and (iv) F2 has infinitely 
many ends. StalUngs' famous ends theorem [38] says that if G is a finitely generated 
group with two or more ends, then G splits nontrivially as an amalgamated product or 
an HNN-extension over a finite subgroup. This can be reformulated in terms of actions 
on trees via Bass-Serre theory [36]. 

A group acts nontrivially on a tree if it has no global fixed-point, i.e., there is no 
node V in the tree with Gv = {v}. A group G is said to split over a subgroup H if 
there is a nontrivial action of G on a tree T such that H is the stabilizer of an edge 
e and the orbit Ge consists of all edges of T. This is equivalent to G splitting as an 
amalgamated product or HNN-extension with H as the amalgamation base, respectively 
the associated subgroup [36]. We shall need the following simple lemma. 



22 



Lemma 12. Let G be a finitely generated group with a nontrivial action on a tree T 
and let H < G be a finite index subgroup. Then H acts nontrivially on T. 

Proof Recall that if 5 is an automorphism of a tree T, then g is said to be elliptic if 
g fixes some point of T. It is well known (this follows immediately from [36, 1.6.4, 
Proposition 25], for instance) that if 5" (n > 1) is elliptic, then g is elliptic. Now if H 
has a global fixed point, then H consists entirely of elliptic automorphisms of T. Let 
[G : H] = n and g G G. Then g" G H and hence is elliptic. It follows that every 
element of G is elliptic. But it is well known [36, 1.6.5, Corollary 3] that any finitely 
generated group of elliptic automorphisms of a tree has a global fixed point, contradict- 
ing that the action of G is nontrivial. It follows that the action of H is nontrivial. □ 

Theorem 6. Let G be a finitely generated group with two or more ends such that the 
intersection of all the finite index subgroups of G is torsion-free. Then G is virtually 
a free product and hence the rational subset membership and submonoid membership 
problems for G are recursively equivalent 

Proof. By StalUngs ends theorem [38], G spUts nontrivially over a finite subgroup. So 
by Bass-Serre theory [36] G acts nontrivially on a tree T so that there is one orbit of 
edges and the stabihzer of an edge is finite. Let H be an edge stabilizer; since H is 
a finite group, by hypothesis there is a normal subgroup N <\ G of finite index such 
Hon = {1}. By Lemma 12 the action of iV on T is nontrivial. Since each edge 
stabihzer in G is a conjugate of H, it follows no element of N \ {1} fixes an edge. 
Therefore, N sphts nontrivially as a free product [36]. This completes the proof. □ 

Corollary 2. Let G be a finitely generated group with two or more ends which is ei- 
ther virtually torsion-free or re sidually finite. Then the rational subset membership and 
submonoid membership problems for G are recursively equivalent. 

Proof. Clearly Theorem 6 appUes under either of these hypotheses. □ 

Let us now come back to graph groups. Theorems 2 and 4 imply that the submonoid 
membership problem is undecidable for every graph group of the form 

G(ru{a},/)~G(r,7)*Z, 

where a ^ S and (S, I) is not a transitive forest. In the rest of the paper, we will 
sharpen this result. We show that for a graph group the submonoid membership problem 
is decidable if and only if the rational subset membership problem is decidable, i.e., if 
and only if the independence alphabet is a transitive forest. In fact, by our previous 
results, it suffices to consider a P4: 

Theorem?. Let E = {a,b,c,d} and I = {{a,b), (6, c), (c, rf)}, i.e, is a P4. 

Then there exists a fixed submonoid M ofG{E, I) such that the membership problem 
ofM within I) is undecidable. 

Proof. We follow the strategy of the proof of Lemma 11, but working in the graph 
group G{E, I) makes the encoding more comphcated. Let R denote the trace rewriting 
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system over the trace monoid M(I7='=^, /) defined in (1), Section 2.3. As usual denote by 
h : (S^^)* — > G{E,I) denote the canonical morphism, which will be identified with 
the canonical morphism h : M(i7='=^, 7) — > G{S, I). Let us fix a finite automaton A 
over the alphabet S"^^ such that the membership problem for h{L{A)) within G{E, I) 
is undecidable; such an automaton exists by Theorem 2. Without loss of generality 
assume that 

A=i{l,...,n},E^\S,qoAqf}), 

where S C {1, . . . , n} x {E^^ U {e}) x {1, . . . , n} and qo ^ qj (since we allow e- 
transitions, we may assume that there is only a single final state qf, which is different 
from the initial state qo). For a state q € {1,. . n}, define the trace q G M.{E^^ , /) by 

q = {adaydiada)-" = {adaY d{a-^ d'^ a'^Y . 

Note that the dependence graph of g is a linear chain. Moreover, every symbol from 
E"^^ is dependent on ad, i.e., does not commute with ad. The following statement is 
straightforward to prove. 

Claim 2. Let gi, . . . , G {1, . . . , n}, £i, . . . , £fe € {1, —1} such that qi / g^+i for all 
l<i<k-l. Then 

NFij(§figf^ • • -9^") = {adaY^d^^{adaY^-'i^ ■ • •d^'=-i(orfa)*-«'=-M^*^(arfa)-«'=. 

Note that this trace starts (resp. ends) with a copy of ada (resp. a~^d~^a~^). 

Let Lp : {E^^Y {E^"^)* be the injective morphism defined by tp{x) = xx for x e 
17^^. Thus, w G i(j4) if and only if <p(w) S ip{L{A)). Since (a;,y) G / implies that 
^p{x) and (^(y) commute, ip can be lifted to an injective morphism ip : M(Z'='=^, /) 
M{E^'^ , I). The reader can easily verify that, for every trace t G M(I7='=^ , J), the equal- 
ity NF ji{(p{t)) = i^(NF R{t)) holds. In particular, (p{t) is irreducible if and only if t is 
irreducible and h{t) = h{u) if and only if h{(fi{t)) = h{ip{u)). 
Let us fix a trace t G M(Z'=^\ /) and define 

A = {qif{x)p-^ I {q,x,p) G (5} C M{E^^,I) and u = qo(p{t)qJ^ G M{E'^\l). 

We wiU show that h{t) G /il-C-lA)) if and only if h{u) G h{A*). 
Let us define a l-cryc/e to be a word in (17='=^)* of the form 

qi(p{vi)q2^ q2(p{v2)qs^ ■ ■ ■ qk-i^{vk-i)%^ qk'fi{vk)qi^ 

such that k > 1, qi, . . . ,qk G {1, . . . , n}, vi, . . . , Vk G (17^^)*, and vi ■ ■ ■ Vk = 1 
in G{E, I) (hence, also 'p{vi) ■ ■ ■ (p{vk) = 1 in G{E, /)). Note that a 1-cycle equals 1 
in G{E, I). We say that a word of the form qi(p{v-i)p^^ q2ip{v2)p2^ • ■ ■ Qm^{vm)Pm^, 
where qi,Pi, ■ ■ ■ ,qm,Pm € {1, . . . ,n} and vi, . . . ,Vm G (17='=^)*, contains a 1-cycle, 
if there are positions 1 < i < j < m such that qi^{vi)p^^ ■ ■ ■ qj(f{vj)p~^ is a 1-cycle. 
If a word does not contain a 1-cycle, then it is called 1-cycle-free. 

Claim 3. Let m > 1 and 

= qiip{vi)p^'^ q2ip{V2)P2^ ■ ■ ■ qmlf{Vm)V^, 
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wheT&qi,pi,...,qm,Pm G {1, ■ . ■ ,n} andvi, . . . ,Vm G iS^^)*.lfv = lmG{E,I), 
then V contains a 1-cycle. 

Proof of Claim 3. We prove Claim 3 by induction over m. Assume that w = 1 in 
G{S, I). If TO = 1, then we obtain the identity 

qMvi)p^^ = {aday^d{ada)-'i^ip{vi){ada)P'd-\ada)-P^ = 1 (10) 

in G{S, I). Assume without loss of generaUty that vi, viewed as a trace, is irreducible 
with respect to R. Then also (p{vi) is irreducible. If (p{vi) = e and pi = qi, then w is a 
1-cycle. If ip{vi) = e, and pi ^ qi, then we obtain a contradiction, since NF/{(gip]"^) 
is nonempty by Claim 2. Now assume that </?(wi) ^ e. In the trace 

{aday'd{ada)-'"-^{vi){adaY^d-^ada)-P' 

only the last a^^ of the factor {a~^d~^a~^)'^^ may cancel against the first a of ip{vi) 
(in case a G miii(L'i)) and the first a of the factor {adaY^ may cancel against the last 
of f{vi) (in case S max(ui)). To see this, note that if a ^ min(wi), then 
{ada)~'^^^p{v\) is irreducible with respect to i?. If a € min(t;i) then ip(vx) = aaip{t) 
for some trace t. Then 

{a-^d-\-^y'ip{vi) = {a-^d-^a-^y'aaip(t) -^r {ar' dr\ir'Y'-^a-^d-\^{t). 

Since a and d do not commute, we cannot have d G min(a(/3(t)), hence cancella- 
tion stops and NF7?,((a^^d^^a^^)''i(p(ui)) = {a^^d^^a^^Ya^^dr^aip{t) where fc = 
gi — 1 > 0. Moreover, if is a maximal symbol of t, then (f{t) = (p{t')a~^a~^ for 
some trace t'. Hence, by making a possible cancellation with the first a in [adaY^, it 
follows finally that 

l<iYR{qnp{vi)p-^^) = {adaY'd{ada)-''a-^d-^xda{adaYd-^{ada)-'" ^ s 

for some trace x, where £ = pi — 1 > 0. This contradicts again (10) and proves the 
inductive base case to = 1 in Claim 3. 
Now assume that to > 2. 

Case 1. There is 1 < i < to such that pi = qi+i. Then ?; = 1 in G{I], I) impUes 

qi^{vi)Pi^ ■ ■ ■ qt-lf{ni-l)Pi^i qi(p{v^V^+i)pll^ qi+2(p{Vi+2)p~+2 ' ' ' 5'm'^(«m)Pm^ 

is 1 in G{S, I). By induction, we can conclude that above word contains a 1-cycle. But 
then also the word v must contain a 1-cycle. 

Case 2. pi ^ gi+i for all 1 < i < to. If there is 1 < i < to such that ?;j = 1 in G{S, I) 
and qi = pi then v contains the 1-cycle q.iip{v.i)pY^ ■ Now assume that qi ^ pi whenever 
= 1 in G(iJ, /). Let v' be the word that results from v by deleting all factors ipi^Vi), 
which are equal 1 in G{E, I). In the following, we consider v' as a trace. Consider a 
maximal factor of v' of the form 

p7%+m+i9i+'2 ■ ■ ■ pjliQj (1 1) 
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where j > / + 1 and Lp{vi+i) — ■ ■ ■ — (p{vj-i) = 1, f{vi) / 1 / in G{S,T). 

Claim 2 show that the i?-normal form of this trace starts (resp. ends) with a copy of ada 
(resp. a~^d~^a~^), and similarly for maximal prefixes (resp. suffixes) of the form 

QiPT^ ■ ■ ■ qi-iPiliQi (resp. p~%+iPi^^ ■ ■ ■ p^nQm). (12) 

In v', factors of the form (11) and (12) are separated by traces fivi), where (p{vi) ^ 1 
in G{U, I). Without loss of generality assume that each such trace f{vi) is irreducible 
and hence non-empty. As for the base case m = 1, one can show that in such a con- 
catenation, only a single noinimal a and a single maximal of a trace ifi{vi) ^ e may 
be cancelled. It follows that NFk(w) ^ e, which contradicts t; = 1 in <G{E, I). This 
concludes the proof of Claim 3. 

Now we can prove h{t) e h{L{A)) if and only if h{u) h{qo(p{t)qJ^) e h{A*). 
First assume that h{t) e h{L{A)). Let ai ■ • • a™ e L{A) such that a^, Qi) G 5 
for 1 < i < m, Qm = qf, and ai - ■ ■am = t in G{S, I). Then 

h{Qoy^{t)Qf^) = M5o'^(ai)5r Vi<p(a2)g^^ • • • qm-i'P{am)qm) e h{A*). 

Now assume that h{qoip{t)qJ^) € h{A*). Thus, 

qo^it)qJ^ = qiv>{ai)pT^q2^ia2)P2^ ■ ■■qmv{am)Pm 

in G{S,I), where qi,pi, . . . ,qjn,Pm G {1, . . . , n}, ai, . . . , G 17=^^ U {e}, and 
{qi, tti, Pi) G 6 ior 1 < i < m. Without loss of generality we may assume that the word 
qiip{ai)pi^q2ip{a2)P2^ ■ • • qm'P{0'm)P^ is 1-cycle-free (otherwise we can remove all 
1-cycles from this word; note that a 1-cycle equals 1 in the group G{S, I)). Let 

Since ?; = 1 in G{S, I), Claim 3 implies that v contains a 1-cycle. We claim that this 1- 

cycle must be the whole word v: first of all, the suffix qi(p{ai)pi^ ■ ■ ■ q„iip{am)Pm^ of 
V is 1-cycle-free. If a prefix qfip{t^^)%^ qi(p{ai)pi^ ■ ■ ■ qi(p{ai)p^^ for i < m is a 1- 
cycle, then g^i+ic/)(a,;+i)p^\ ■ • • q„iif{arn)p^ = 1 in G{IJ, I). Hence, Claim 3 implies 
that the word (j,;+i99(ai+i)p!^\ ■ • • q,„(f{a,n)Pm^ contains a 1-cycle, contradicting the 
fact that the word qiip{ai)pi^ ■ ■ ■ qm'^{am)p^ is 1-cycle-free. Thus, indeed, is a 1- 
cycle. Hence, go = qi, qf = Pm, Pi = qi+i for 1 < i < m, and t~^ai ■ ■ - am = 1 
in G{S,I), i.e., h{t) = h{ai ■ ■ ■a,,,) G h{L{A)). This shows that the membership 
problem for the submonoid h{A*) within G{IJ, I) is indeed undecidable. □ 

Recall that a graph is not a transitive forest if and only if it either contains an induced 
C4 or P4 [39]. Together with Mihailova's result for the generaUzed word problem of 
F{{a, b}) X F{{c, d}), Theorems 2 and 7 imply: 

Corollary 3. The submonoid membership problem for a graph group G{S, I) is de- 
cidable if and only if {E, I) is a transitive forest. Moreover, if{E,I) is not a transitive 
forest, then there exists a fixed submonoid M of G{IJ, I) such that the membership 
problem for M within G{S, I) is undecidable. 

Since P4 is a chordal graph, the generalized word problem for G{PA) is decidable 
[25]. Hence, G(P4) is an example of a group for which the generahzed word problem 
is decidable but the submonoid membership problem is undecidable. 
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5 Open problems 



The definition of the class C at the beginning of Section 3 leads to the question whether 
decidability of the rational subset membership problem is preserved under direct prod- 
ucts with Z. An affirmative answer would lead in combination with the results from [24, 
31] to a more direct proof of Theorem 1. 

Concerning graph groups, the precise borderline for the decidability of the general- 
ized word problem remains open. By [25], the generalized word problem is decidable if 
the independence alphabet is chordal. Since every transitive forest is chordal. Theorem 2 
does not add any new decidable cases. On the other hand, if the independence alphabet 
contains an induced C4, then the generalized word problem is undecidable [30]. But it 
is open for instance, whether for a cycle of length 5 the corresponding graph group has 
a decidable generalized word problem. 

Another open problem concerns the complexity of the rational subset membership 
problem for graph groups, where the independence alphabet is a transitive forest. If 
the independence alphabet is part of the input, then our decision procedure does not 
yield an elementary algorithm, i.e., an algorithm where the running time is bounded 
by an exponent tower of fixed height. This is due to the fact that each calculation of 
the Parikh image of a context-free language leads to an exponential blow-up in the size 
of the semilinear sets in the proof of Lemma 7. An NP lower bound follows from the 
NP-completeness of integer programming. 

Theorem 4 and 5 lead to various research directions. One might try to get rid of 
the restriction that g^^Ag n A = 1 = h~^Ah n A for some g £ G, h G H and the 
analogous restrictions for HNN extensions. These two results together would imply that 
Corollary 2 holds for all groups with two or more ends. 

In fact it is natural to ask whether, for every finitely generated group G, the sub- 
monoid membership and rational subset membership problems are recursively equiva- 
lent. By Theorem 4, this is equivalent to the preservation of the decidability of the sub- 
monoid membership problem under free products (which is again not known to hold): 
simply choose for H in Theorem 4 any nontrivial group with a decidable rational subset 
membership problem. Recall that the decidability of the generalized word problem as 
well as the rational subset membership problem is preserved under free products. No- 
tice that for a torsion group, the submonoid membership problem is equivalent to the 
generaUzed word problem, while the rational subset membership problem reduces to 
membership in products H\ - ■ Hn of finitely generated subgroups. 
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